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ABSTRACT 

Special Competition Bilingual Enrichment Academic 
Russian Program is a federally-funded program that served 623 native 
Russian-speaking, limited-English-proficient (LEP) students in nine 
public and two private high schools in New York City in 1992-93, its 
first year of operation. Students received instruction in English as 
a second language (ESL) , native language arts (NLA), mathematics, 
science, social studies, and business skills. Staff development and 
parental involvement and educational activities were also important 
program components. The program met all objectives in ESL, NLA, 
student performance on state competency and achievement tests, 
mathematics, science, social studies, business, career awareness, 
attitudes toward the Russian language, attendance, and parental 
involvement. Recommendations for program improvement include: 
exploration of additional techniques to increase student subject area 
skills at selected sites, possibly by augmenting peer and 
paraprof ess ional tutoring and through computer instruction; 
development of ties with business and community organizations and 
enlist their support; and holding parent involvement activities at 
locations and times more convenient for parents. (Author/MSE) 
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EXECUTIVE SUMMARY 



Special Competition Bilingual Enrichment Academic Russian Program was an 
Elementary and Secondary Education Act (E.S.E.A.) Title Vll-funded project in its first 
year of operation in 1992*93. The project functioned at nine public high schools 
(Abraham Lincoln, Franklin D. Roosevelt, James Madison, New Utrecht, Lafayette, 
Edward R. Murrow, and South Shore High Schools in Brooklyn and Forest Hills and 
Richmond Hill High Schools in Queens) and two private high schools (Harama and 
Be'er Hagola Yeshivas in Brooklyn). The project served a total of 623 students of 
limited English proficiency (LEP). Participating students received instruction in 
English as a second language (E.S.L); native language arts (N.LA.); and the content 
areas of mathematics, science, social studies, and business skills. 

Teachers of participating students had the opportunity to attend monthly staff 
development workshops. 

The project sought to establish an active parental component, which included 
bilingual workshops and courses in bilingual education at the City University of New 
York (CUNY), but found it difficult to elicit participation. 

The project met all its objectives: E.S.L; Russian N.LA.; student performance 
on Regents Competency Tests (R.C.T.s) and Regents examinations; the content area 
subjects of mathematics, science, social studies, and business; career awareness; 
attitude toward Russian language; attendance; and parental involvement. 

The conclusions, based on the findings of this evaluation lead to the following 
recommendations: 

• Explore additional techniques to increase students' acquisition of skills in 
mathematics, science, and social studies at those sites where a smaller 
percentage than expected passed their courses in these areas. Consider 
augmenting a peer- and paraprofessionaMutoring program to work with 
students during study halls or outside school hours. Consider also 
initiating a program of computer-assisted instruction. 

• Explore ways to develop ties with business and community organizations 
and enlist their support in t5~s career program component as stated in the 
proposal. 

• Consider holding some parental involvement activities at sites closer to the 
homes of potential participants and at times more convenient to their 
schedules. 
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I. INTRODUCTION 

This report documents the Office of Research, Evaluation, and Assessment's 
(OREA's) evaluation of the Elementary and Secondary Education act (E.S.E.A.) Title 
Vll-funded project, Special Competition Bilingual Enrichment Academic Russian 
Program. 

PROJECT CONTEXT 

The program served Russian-speaking students of limited English proficiency 
(LEP) at nine public and two private high schools in Brooklyn and Queens. The 
public schools were Abraham Lincoln, Franklin D. Roosevelt, Madison, New Utrecht, 
Lafayette, Edward R. Murrow, and South Shore High Schools in Brooklyn and Forest 
Hills and Richmond Hill High Schools in Queens. The two private schools were 
Harama and Ber'er Hagola Yeshivas in Brooklyn. It is estimated that more than 50 
percent of all the Russians immigrating to the United Sates live in New York City, and 
over 75 percent of them live in Brooklyn and Queens, where the project sites are 
located. 

Of 2,317 students registered at Abraham Lincoln High School during the 1992- 
93 school year, 45.0 percent (1,043) were European- American, 26.5 percent (614) 
were African-American, 19.0 percent (440) were Latino, and 10 percent (219) were 
Asian-American* Almost a quarter (24 percent) were LEP, and nearly half (45 
percent) came from low-income families. 



♦Percentage may not total 100 because of rounding or ambiguous data. 
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Of the 3,528 students registered at Franklin D. Roosevelt High School, 43.7 
percent (1,542) were European-American, 28.7 percent (1,014) were Latino, 19.6 
percent (690) were Asian-American, and 7.8 percent (276) were African-American. 
Of these students, 30 percent were LEP, and 35 percent came from low-income 
families.* 

Of the 2,846 students enrolled at James Madison High School, 48.6 percent 
(1,383) were European-American, 31.1 percent (884) were African-American, 11.5 
percent (328) were Latino ; and 8.6 percent (246) were Asian-American.* Fifteen 
percent were LEP and 25 percent came from low-income families. 

Of a total of 2,674 students enrolled at New Utrecht High School, 53.4 percent 
(1,428) were European-American, 20.6 percent (551) were African-American, 13.4 
percent (359) were Latino, and 11.1 percent (298) were Asian-American* Of these 
students, 13 percent were LEP and 25 percent came from low-income households. 

At Lafayette High School, 2,680 students were registered for the year under 
review. Of those, 35 percent (939) were African-American, 30.5 percent (818) were 
European-American, 19.1 percent (513) were Latino, 14.1 percent (406) were Asian- 
American, and 0.1 percent (4) were Native American.* Sixteen percent (438) were 
LEP and 20 percent came from low-income families. 

Of the 3,216 students registered at South Shore High School in the year under 
review, 46.4 percent (1,493) were African-American, 43.1 percent (1,387) were 



♦Percentage may not total 100 because of rounding or ambiguous data 
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European-American, 7.4 percent (237) were Latino, 3.0 percent (95) were Asian- 
American, and 0.1 percent (4) were Native-American. 

Of the 3,409 students at Edward Murrow High School, 46.7 percent (1,592) 
were European-American, 23.7 percent (808) were African-American, 15.6 percent 
(533) were Latino, and 14.0 percent (476) were Asian-American. Eleven percent of 
the student population was LEP, and 20 percent came from low-income families. 

Of the 2,911 students at Forest Hills High School, 47.1 percent (1,371) were 
European-American, 23.5 percent (683) were Asian-American, 19.2 percent (558) 
were Latino, and 10.0 percent (291) were African-American..* Fourteen percent 
were LEP and 16 percent came from low-income families. 

Of the 2,417 students registered at Richmond Hill High School, 37.2 percent 
(898) were Latino, 24.2 percent (578) were European-American, 21 .0 percent (508) 
were Asian-American, and 14.0 percent (338) were African-American.* Twelve 
percent of the students were LEP and 34 percent came from low-income families. 

An OREA evaluator visited Franklin D. Roosevelt High School in Brooklyn. The 
building was large and in good condition. The classrooms were bright and 
spacious, and student work was displayed on bulletin boards. Although the halls 
were crowded, students were orderly. An OREA consultant also visited Abraham 
Lincoln High School. Posters, maps, and student displays were exhibited on the 
walls. The school was well maintained, and the classrooms were roomy and cheerful. 



♦Percentages may not total 100 because of rounding or ambiguous data. 
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STUDENT CHARACTERISTICS 

The project served a total of 623 LEP students in grades nine through twelve. 
(See Table 1 .) Students were classified as LEP if they scored at or below the 40th 
percentile on the Language Assessment Battery (LAB). All students had Russian as 
their native language. Male students numbered 300 (48.2 percent), female 323 (51 .8 
percent). 

All of the participating students were born in former Soviet Union. Most (95.7 
percent) came from low-income families and were eligible for the free-lunch program. 

TABLE 1 



Number of Students in Special Competition Bilingual 
Enrichment Academic Russian Program, by Site 



| Site 


9 


10 


11 


12 


Total | 


I Abraham Lincoln 


31 


19 






50 1 


Franklin D.Roosevelt 


42 


18 






60 


James Madison 


22 


9 






31 


New Utrecht 


37 


34 


28 


7 


106 


Lafayette 


26 


19 


23 


19 


87 


South Shore 


7 


3 






10 


Edward Murrow 


55 


47 


27 


20 


149 


Forest Hills 


18 


32 






50 


Richmond Hill 


2 


4 


5 


6 


17 


Harama 


13 


1 






14 


I Be'er Hagola 


36 


12 


1 




49 


| Total 


289 


198 


84 


52 


623 
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Needs Assessment 

Before instituting this program, the principals of the selected schools conducted 
a needs assessment of the targeted LEP students, their families, and the staff who 
would serve them. The data obtained from this study indicated three primary needs: 
(1) to assist LEP students in achieving competence in English while mastering 
subjects such as mathematics and science to help them graduate; (2) to offer 
students the skills necessary for gainful employment, and (3) to involve parents of 
participating students in the education of their children, thus helping to insure their 
success. 



PROJECT OBJECTIVES 
Student Objectives 

• Seventy-five percent of the participating students will improve their English 
language skills in understanding, reading, writing, and speaking, as 
measured by the Language Assessment Battery. 

• Seventy-five percent of the enrolled students will demonstrate significant 
achievement in Russian by passing teacher-made tests. 

• Seventy-five percent of the project students who take the Regents 
Competency test will pass the Regents exam in Russian. 

• Eighty-five percent of the students will pass each subject in the content 
areas of mathematics, science, and social studies at the end of each 
semester, securing 65 percent or more as measured by teacher-made tests. 

• In e< ch of the content areas of mathematics, social science, and social 
studies, 75 percent of the project students who take the examination will 
pass the Regents Competency Test. 

• Seventy-five of the project students will secure a passing grade of 65 or 
more in business subjects. 
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• Seventy-five percent of the participating students will demonstrate 
awareness of career choices and professional requirements as measured 
by a student assessment form. 

• Seventy-five percent of the students will develop a more positive attitude 
toward the Russian language. 

• The attendance rate of project students will be equal to that of the school 
as a whole. 

• At least 75 percent of the project students who are enrolled in their senior 
year will graduate from high school at the end of the year. 

Parent Involvement Objectives 

• Seventy percent of parents of participating Russian LEP students will have 
an improved involvement and understanding of their children's educational 
needs and career goals as result of participating in advisory council 
meetings, bilingual workshops, City University of New York bilingual 
education courses, or individual advisement sessions offered by project 
staff. 

• At least five parents of Russian LEP students "will take some of the six 
college credits required in order to serve as bilingual paraprofessionals. 

• At least 30 parents and/or teachers, as measured by attendance records, 
will attend at least one of three workshops the project will offer on careers 
in bilingual education. 



PROJECT IMPLEMENTATION 

During the 1992-93 school year, the project provided instructional and support 
services to 623 Russian-speaking students and their families. The project's main 
goal was to promote the acquisition of language and content area skills by LEP 
students while introducing the skills necessary for gainful employment. 

Abraham Lincoln, Franklin D. Roosevelt, and Forest Hills High Schools offered 
content area subjects 70 percent of the time in Russian and 30 percent of the time in 
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English. At James Madison, New Utrecht, Lafayette, Edward R. Murrow, and South 
Onore High Schools, the content areas were taught 75 percent of the time in Russian 
and 25 percent of the time in English. 

The project offered parental involvement activities such as advisory council 
meetings, a workshop on AIDS education, trips, and food festivals. The project also 
provided reimbursement to three parents for college credits. 

In-service staff development activities such as conferences, seminars, and 
workshops were concerned with the Russian bilingual curriculum and curriculum 
development for AIDS education. 
Materials Methods, and Techniques 

The project offered English as a second language (E.S.L) at beginning and 
intermediate levels, and native language arts (N.LA.) at intermediate and advanced 
levels. At the beginning of the year, the teachers emphasized bilingual instructional 
methodologies in the content areas: the material was first presented in Russian and 
then repeated in English. This enabled students to acquire concepts and practice 
emerging skills in English. As the year progressed, teachers made the transition to 
greater use of English instruction using E.S.L methodologies. 

Teachers of participating students used a variety of strategies and techniques, 
including cooperative learning, peer tutoring, and content-based and whole language 
approaches. 

In N.LA. classes, students saw films and attended plays, recit <!s, and lectures 
presented by a variety of performers and by professors of Russian literature. 

7 
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Project participants from Lincoln, Madison, Franklin D. Roosevelt, South Shore, 
and Forest Hills High Schools participated in the 11th Annual Russian Essay 
Competition and the Bilingual Oratory Olympics in Russian. All nine participants 
from Franklin D. Roosevelt High School won medals, including gold and silver 
medals at the Russian Language Olympiada and gold and bronze medals in the 
second round of the Russian contest held in Moscow. Participants from New 
Utrecht, South Shore, and Forest Hills High Schools also excelled. 

Participants interested in education and the teaching profession were offered 
the opportunity to intern at P.S. 100 in Queens. Students from Abraham Lincoln 
High School also taught Russian to elementary classes as part of the career 
education component. 

Project students traveled to the New York City Job and Career Center to 
explore career choices and opportunities. At Wngsborough Community College, 
students were offered an orientation in career choices and attended a special 
presentation by the U.S. Marine and Fisheries Department on career opportunities. 
Madison High School held an orientation session for project parents and students on 
career opportunities in medicine, law, and computers. The project also invited 
guests to speak on science topics. 

For a list of instructional materials used in the project, please see Appendix A. 
Capacity Building 

Next year, tax-levy funds will assume the cost of curriculum development, 
training sessions for staff and parents, field trips, and films. 
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Staff Qualification* 

Title VII staff. The project director, one resource specialist, three 
paraprofessionals, and a secretary were fully funded by Title VII. Another resource 
specialist was partially funded by Title VII and partially by Chapter I tax-levy funds. 
For a description of staff degrees and language proficiencies (teaching or 
communicative*), see Table 2. 



TABLE 2 
Project Staff Qualifications 



| Position Title 


Degree(s) 


Language I 
Competence H 


Project Director 


M.A. 


Russian TP | 
Spanish TP 
Polish TP 
Yiddish TP | 


Resource teacher 
1 guidance 


M.A. 


Russian TP 9 
Ukrainian CP 1 
Yiddish CP 


1 Resource specialist 


B.A. 


Russian TP 8 


Paraprofessional 


B.A. 


Russian TP J 


Paraprofessional 


B.A. 


Russian TP 


Paraprofessional 


B.A. 


Russian TP 


Secretary 


A.A.S. 





♦Teaching proficiency (TP is defined as the ability to use LEP students' native 
language in teaching language arts of other academic subjects. Communicative 
proficiency (CP) is defined as a non-native speaker's basic ability to communicate 
and interact with students in their native language. 

9 
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The project director's responsibilities included supervising and coordinating 
activities, selecting and training staff, and providing OREA with evaluation data. 

The resource specialists' responsibilities were to assist classroom teachers in 
the development and implementation of instructional activities, provide in-service 
training to teachers, help plan activities for parents of participating students, and 
select instructional material. The resource specialists also acted as advisors to each 
school's Bilingual Parent Advisory Committee (BPAC), and provided guidance and 
career counseling to the target student population. 

The guidance teacher and the paraprofessionals worked with students 
individually or in groups as the need arose. 

Other staff . Tax-levy funds paid the salaries of the 20 classroom teachers and 
one bilingual guidance counselor who provided services to project students. (For 
qualifications, please see Table 3). 

All teachers taught in the subject areas in which they were certified. 

TABLE 3 

Qualifications of Other Staff Serving Project Students 



Title 


Degree(s) 


Certification 


Language 
Competence 


20 Teachers 


4 M.A. 
11 BA 

5 B.S. 


5 E.S.L 
16 Russian 
3 Biology 
3 Social Studies 
3 Mathematics 


20 Russian TP 


1 Guidance Counselor 


M.A. 


Bit. Guidance Counselor 


Russian TP 
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Staff Development . Teachers and project staff participated in a series of 
monthly workshops on staff responsibilities and curriculum development. Two staff 
members took college courses. 
Instructional Time Spent on Particular Tasks 

See Appendix B for examples of class schedules. 
Lenoth of Time Participants Rece ived Instruction 

Students had a mean of 8.5 years (s.d.=1 .1) of education in a non-English- 
speaking school system and 1.4 years (s.d.=0.8) of education in the United States. 
The median time students participated in the project was 9 months. 
Activities to improve Pre-referraJ Evaluation Proce dures for exceptional Children 

Teachers referred those students thought to be in need of special education 
services to the School-Based Support Team (S.B.S.T.) for evaluation. Russian bilingual 
teachers translated as necessary. The two parochial schools referred students thought 
to be in need of special education to the Board of Education of the City of New York. 
Gifted and talented students were identified by teacher judgment and course grades. 
Instructional Services for Students with Special Needs 

The project offered individual tutoring by teachers and paraprofessionals as well 
as peer tutoring. Students were encouraged to participate in extracurricular activities 
and to take advanced placement courses in the mainstream. They also were given 
special assignments in creative writing and were offered special science courses. 
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PARENT AND COMMUNITY INVOLVEMENT ACTIVITIES 

The parents of participating students were involved in a number of project- 
sponsored activities, such as workshops, a Parents' Institute, a conference given by the 
New York State Association for Bilingual Education (SABE), E.S.L classes, and field 
trips. Three parents from Forest Hills, Abraham Lincoln, and Franklin D. Roosevelt High 
Schools took courses at Adelphi College in order to serve as paraprofessionals. Their 
tuition was paid by the project. 




II. EVALUATION METHODOLOGY 



FVALUATION DESIGN 

Project Group's Educational Progress as Compared to Tha t of an Appropriate 
Non-Proiect Group 

OREA used a gap reduction design to evaluate the effect of language 

instruction on project students' performance on standardized tests. Because of the 

difficulty in finding a valid comparison group, OREA used instead the groups on 

which the tests were normed. Test scores are reported in Normal Curve Equivalents 

(N.C.E.s), which are normalized standard scores with a mean of 50 and a standard 

deviation of 21.1. It is assumed that the norm group has a zero gain in N.C.E.s in 

the absence of supplementary instruction and that participating students' gains are 

attributable to project services. 

Applicability of Conclusions to All P ersons Served bv Project 

Data were collected from all participating students for whom there were pre- 
and posttest scores. (There were no pretest data on students who entered the 
program late; therefore, posttest data for them will serve as pretest data for the 
following year.) Instruments used to measure educational progress were appropriate 
for the students involved. The LAB, the New York State Regents in Russian, and the 
Regents Competency Test (R.C.T.) are used throughout New York City to assess the 
growth of English, Russian, mathematics, science, and social studies skills in 
populations similar to those served by the Special Competition Bilingual Enrichment 
Academic Russian Program. 
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INSTRUMENTS OF MEASUREMENT 

OREA compared pre- and posttest scores on the LAB to assess the E.S.L 

objective, and examined grades on the Russian Regents to assess the N.LA. 

objective. To assess the objective for content area subjects, OREA used R.C.T. 

scores and final course grades, as specified in the objective. 

All students were tested at the appropriate grade level. The languages of the 

LAB and Russian R.C.T. were determined by the tests themselves, whereas the 

language of the R.C.T. was determined by the language in which the students 

received instruction in mathematics, science, and social studies. 

According to the publishers* test manuals, all standardized tests used to gauge 
project students' progress are valid and reliable. Evidence supporting both content 
and construct validity is available for the LAB. Content validity is confirmed by an 
item-objective match and includes grade-by-grade item difficulties, correlations 
between subtests, and the relationship between the performance of students who are 
native speakers of English and students who are LEP. To support reliability, the 
Kuder-Richardson Formula 20 (KR20) coefficients and standard errors of 
measurement (SEM) are reported by grade and by form for each subtest and total 
test. Grade reliability coefficients, based on the performance of LEP students on the 
English version, ranged from .88 to .96 for individual subtests and from .95 to .98 for 
the total test. 

To assess the growth of cultural pride, OREA developed a Likert-type 
questionnaire which project personnel administered to all participating students (See 
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Appendix C.) A Likert scale typically employs five choices expressing different 
degrees of agreement or disagreement. 

In order to assess the career education/exploration objectives, project 
personnel distributed an OREA-developed Likert-type questionnaire (see Appendix 

C). 

Another OREA-developed Likert-type questionnaire was used to assess the 
parental involvement objective for improved understanding of their child's 
educational needs and goals. (See Appendix C). 

DATA COLLATION AND ANALYSIS 
Data Collection 

To gather qualitative data, an CREA evaluation consultant carried out on-site 
and telephone interviews with the project director several times during the school 
year and also observed two classes on each of two visits. The project evaluator 
collected the data and prepared the final evaluation report in accordance with the 
New York State E.S.E.A. Title VII Bilingual Education Final Evaluation Report format, 
which was adapted from a checklist developed by the staff of the Evaluation 
Assistance Center (EAC) East in consultation with the Office of Bilingual Education 
and Minority Language Affairs (OBEMLA). 
Pmpar Administratio n 9 f Instruments 

Qualified personnel received training in testing procedures and administered 
the tests. Test administrators followed guidelines set forth in the manuals 
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accompanying standardized tests. Time limits for subtests were adhered to; 
directions were given exactly as presented in the manual. 
Testing at Twelve-Month Intervals 

Standardized tests were given at 12-month intervals, following published 
norming dates. 
Data Analysis 

Accurate scoring and transcription of results . Scoring, score conversions, and 
data processing were accomplished electronically by the Scan Center of the Board 
of Education of the City of New York. Data provided by the Scan Center were 
analyzed in the Bilingual, Multicultural, and Early Childhood Evaluation Unit of OREA. 
Data collectors, processors, and analysts were unbiased and had no vested interest 
in the success of the project. 

Use of analyses and reporting procedures appropriate for obtained data To 
determine the proportion of students showing gains in English-language proficiency, 
OREA computed the percentage of students with pretest/posttest gains on the LAB. 
To assess the significance of students' achievement in English, OREA computed a 
correlated f-test on the LAB N.C.E. scores. The Mast determined whether the 
difference between the pre- and posttest scores was significantly greater than would 
be expected from chance variation alone. 

The only possible threat to the validity of any of the above instruments might be 
that LAB norms were based on the performance of English proficient (EP) rather than 
LEP students. Since OREA was examining gains, however, this threat was 
inconsequential— the choice of norming groups should not affect the existence 
of gains. 
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PARTICIPANTS' EDUCATIONAL PROGRESS 

The project carried out all instructional activities outlined in its original design. 
Participants' Progress i n English 

Throughout the school year, students had ample opportunity to develop their 

English language skills. 

An OREA consultant observed two beginning level E.S.L. classes. Teachers 
used drill and repetition in both classes. Students volunteered responses, although 
it was not clear that all the students understood all the questions the teacher was 
asking; there were many different levels of comprehension and competence among 
the students. No paraprofessional was present in the first class observed. The 
second class had a paraprofessional, who distributed materials and explained the 
instructions in English. The students in this class were divided into five groups. The 
teacher distributed a photocopied passage and pictures to each group and asked 
the students to arrange the pictures in the correct sequence according to the 
passage. 

The evaluation objective for English as a second language was: 

• Seventy-five percent of the participating students will improve their English 
language skills in understanding, reading, writing, and speaking, as 
measured by the Language Assessment Battery. 

There were complete pre- and posttest scores on the LAB for 350 students from 
grades nine through twelve. (See Table 3.) The mean gain for these students was 
16.1 N.C.E.S. This was an extremely large gain and was statistically significant 
(p<.05). 
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Overall, 78.9 percent the students for whom data were available demonstrated 
an increase from pretest to posttest. 

The project met its objective for English as a second language. 
Participants' Progress in Native Language Arts 

South Shore and Richmond Hill High Schools and Harama Yeshiva did not 
have a Russian bilingual teacher for content area instruction, since the total number 
of students in any one grade was less than 20. A Russian bilingual paraprofessional 
rendered part-time service (two days a week). This paraprofessional served project 
students at Lafayette High School the rest of the time. 

The OREA consultant observed a Russian literature lesson based on a 
biography of Leo Tolstoy. No paraprofessional was present in the class. The role- 
played a section of the biography. They worked on pairs, with one student 
portraying a reporter interviewing Tolstoy and the other portraying the author. The 
students had also drawn pictures of Tolstoy's living room, based on descriptions of it 
in the biography. The lesson was stimulating, and students were very involved. 

The project proposed two N.LA. objectives: 

• Seventy-five percent of the enrolled students will demonstrate significant 
achievement in Russian as measured by teacher-made tests. 

More than 90 percent of the students passed their Russian N.LA. courses both 

semesters, although the percentage of those passing decreased from 94.8 in the fall 

to 90.7 in the spring. (See Table 5.) The project director reported that peer tutoring 
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and additional small group instruction by Russian bilingual paraprofessionals will be 
provided next year, particularly at those sites without bilingual content area teachers. 
The project met its N.LA. objective for earning passing grades in N.L.A. 



TABLE 5 

Passing Grades in Native Language Arts, by Site 





rcsii 


Spring 


School 


Number 
Enrolled 


Percent 
Passing 


Number 
Enrolled 


Percent 
Passing 


1 Abraham Lincoln 


7 


100.0 


— 


— 


| Franklin D.Roosevelt 


26 


92.3 


17 


94.1 


| James Madison 


4 


100.0 


3 


100.0 


1 New Utrecht 


64 


96.9 


70 


92.9 


| Lafayette 


78 


87.2 


72 


75.0 


South Shore 










Edward Murrow 


80 


97.5 


68 


98.5 


Forest Hills 


50 


100.0 


49 


98.0 


Richmond Hill 










Harama 










Be'er Hagola 










Total 


309 


94.8 


279 


90.7 



• At least 75 percent of participating students passed their N.LA. 
classes at all reporting sites. 
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• Seventy-five percent of the project students Mio take the Regents test will 
pass the Regents exam in Russian. 

The project reported scores for 54 students who took the Russian Regents 
examination. All passed. 

The project met its N.L.A. objective for passing the Russian Regents examination. 
LEP Participants' Academic Achievement 

At the beginning of the year, the teachers emphasized bilingual instructional 
methodologies in the content areas. The material was first presented in Russian, 
and the students gave the corresponding words in English. This enabled them to 
acquire concepts and practice emerging English skills. As the year progressed, 
teachers made the transition to greater use of English in instruction through E.S.L 
methodologies. 

Teachers of participating students used many of strategies and techniques, 
including cooperative learning, peer tutoring, and content-based and whole language 
approaches. 

An OREA consultant observed a tenth-grade social studies class. The teacher 
distributed a photocopied passage she had translated from English into Russian 
entitled "Is the Relationship Between War and Technology Always Beneficial to 
Humanity?" The lesson was to build vocabulary. 

The project proposed four objectives for content area subjects: 

• Eighty-five percent of the students will pass each subject in the 

content areas of mathematics, science, and social studies at the end of 
each semester, securing 65 percent or more as measured by the school 
final exam. 
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Overall, in all subject areas and at all sites, more than 85 percent of the 
students received passing grades. (See Tables 6a,b, and c.) 

The project met its objective overall, but failed to meet it at Lafayette and South 
Shore High Schools in mathematics, science, and social studies; at Abraham Lincoln 
and Franklin D. Roosevelt High Schools in mathematics; at Richmond Hill and New 
Utrecht High Schools in science; and at Richmond Hill High School in social studies. 

The project met its final course grade content area objective. 

TABLE 6a 



Passing Grades in Content Area Courses 
(Mathematics), by Site 





Fall 


Spring | 


S School 


Number 
Enrolled 


Percent 
Passing 


Number 
Enrolled 


Percent 1 
Passing § 


Abraham Lincoln 


42 


100.0 


47 


78.7 


Franklin D.Roosevelt 


58 


84.5 


55 


83.6 


James Madison 


31 


83.9 


30 


86.7 


New Utrecht 


88 


77.3 


92 


87.0 


Lafayette 


70 


72.9 


72 


76.4 


South Shore 


10 


60.0 


9 


66.7 


Edward Murrow 


138 


95.7 


129 


99.2 


Forest Hills 


50 


94.0 


48 


91.7 


Richmond HIM 


14 


92.9 


14 


85.7 


Harama 


14 


85.7 


14 


92.9 


Be er Hagola 


48 


87.5 


47 


91.5 


Total 


563 


86.7 


557 


88.0 J 



• Except at three high schools in the fall and three in the spring, over 
85 percent of project students passed their content area courses in 
mathematics. 
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TABLE 6b 

Passing Grades in Content Area Courses 
(Science), by Site 





Fall 


Spring j 


School 


Number 
Enrolled 


Percent 
Passing 


Number 
enrolled 


n 

Percent i 
rossing 


1 Abraham Lincoln 


39 


92.3 


33 


90.9 


1 Franklin D.Roosevelt 


54 


88.9 


54 


87.0 


James Madison 


28 


96.4 


30 


93.3 


New Utrecht 


61 


90.2 


78 


79.5 


Lafayette 


39 


66.7 


51 


74.5 I 


South Shore 


10 


50.0 


10 


70.0 I 


Edward Murrow 


140 


91.4 


130 


100.0 


| Forest Hills 


50 


78.0 


49 


91.8 


J Richmond Hill 


7 


57.1 


9 


77.8 


| Harama 


14 


92.9 


14 


92.9 | 


1 Be'er Hagola 


49 


93.9 


49 


93.9 | 


l Total 


491 


87.0 


507 


89.3 | 



• At eight sites in both the fall and the spring, over 85 percent of 
participating students passad science. 
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TABLE 6c 



Passing Grades in Content Area Courses 
(Social Studies), by Site 





School 


Fall 


Spring 


Number 
Enrolled 


Percent 
Passing 


Numoer 
Enrolled 


Passing 


Abraham Lincoln 


47 


95.7 


43 


97.7 


Franklin D.Roosevelt 


53 


79.2 


55 


94.5 


James Madison 


29 


89.7 


30 


93.3 


| New Utrecht 


80 


90.0 


97 


89.7 


1 Lafayette 


73 


76.7 


74 


75.7 | 


1 South Shore 


10 


60.0 


10 


70.0 


Edward Murrow 


138 


92.0 


136 


100.0 


Forest Hills 


50 


92.0 


49 


91.8 


Richmond Hill 


14 


64.3 


17 


82.4 


Harama 


14 


78.6 


14 


92.9 


Be'er Hagola 


47 


95.7 


45 


95.6 


I Total 


555 


87.4 


570 


91.8 



• At least 85 percent of students passed social studies at more sites in 
the spring than in fall. 
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• Seventy-five percent of the project students who take the exam will pass 
the Regents Competency Test in the content areas of mathematics, 
science, and social studies. 

The project reported scores for 43 students who took the R.C.T. in 
mathematics. Of those, 38 (88.4 percent) passed. Twenty-four project students took 
the R.C.T. in science and 18 (75 percent) passed. In social studies, the project 
reported R.C.T. scores for 14 students, of whom 13 (92.9 percent) passed. 

The project met its content area objective for receiving passing grades 
on the R.C.T. 

• Seventy-five of the project students will secure a passing grade of 65 
or more in business subjects. 

More than 75 percent of the students enrolled in the business subjects of 

business skills, keyboarding, and computers received passing grades of at least 65. 

(See Tables 6d, e, and f.) The only negative aspect of this component of the project 

was that it did not provide students with sufficient exposure to the real world of 

business. 

The project met its objective for business skill subjects. 
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TABLE 6d 



Passing Grades in Content Area Courses 
(Business Skills), by Site 



School 



Abraham Lincoln 



Fall 



Number 
Enrolled 



Percent 
Passing 



Spring 



Number 
Enrolled 



Percent 
Passing 



Franklin D.Roosevelt 



James Madison 



100.0 



100.0 



13 



92.3 



100.0 



New Utrecht 



100.0 



1 Lafayette 



8 



75.0 



88.9 



South Shore 



Edward Murrow 



87 



95.4 



22 



100.0 



Forest Hills 



Richmond Hill 



100.0 



11 



90.9 



Harama 



Be'er Hagola 



45 



100.0 



45 



100.0 



Total 



156 



96.2 



114 



97.4 



• In both semesters, at least 75 percent of students enrolled in 
business skills classes at all sites passed their courses. 



TABLE 6e 



Passing Grades in Content Area Courses 
(Keyboarding), by Site 





Fall 


Spring 


ocnooi 


Number 
enrolled 


Percent 


Number 

PnrnllArl 


Percent 
Passina 


Aurcuiarn uncoin 






0 


inn o 


□ \\cu I Mil 1 L/. nvvoo vol l 


8 


87.5 


Q 

o 


IUU.U 


1 UcUIIUo iVtCUJIOwl 1 






o 
o 


OO.f 


1 Nio\A/ 1 ftrooht 


0 


inn n 


ft 

w 


83 3 1 


Lafavett© 


6 


83.3 


5 


60.0 


South Shojre 










Edward Murrow 


1 


100.0 


34 


100.0 


| Forest Hills 










1 Richmond Hill 


3 


100.0 


6 


66.7 1 


1 Harama 










\ Be'er Hagola 


6 


100.0 


5 


100.0 j 


1 Total 


29 


93.1 


72 


91.7 | 



• In the fall, over 75 percent of students at all reporting sites passed 
their courses in keyboarding. In the spring, this was achieved at five 
of the eight reporting sites. 
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TABLE 6f 



Passing Grades in Content Area Courses 
(Computers), by Site 





Fall 


Spring | 


School 


Number 
Enrolled 


Percent 
Passing 


Number 
enrolled 


Percent I 


U ADranam uncoin 


2 


100.0 




~ 1 


g rranKiin u.nooseveii 











1 james Mauison 


4 


100.0 


1 


100.0 (I 


isew uirscnt 


3 






1 

— - — 


LSUayotto 


3 


66.7 


3 


33.3 


South Shore 










Edward Murrow 


20 


100.0 


4 


100.0 


Forest Hills 










Richmond Hill 










Harama 


13 


76.9 


14 


100.0 I 


1 Be'er Hagola 


3 


100.0 


5 


100.0 | 


1 Total 


48 


91.7 


27 


92.6 | 



• Except at one site, over 75 percent of students passed their 
computer courses both semesters. 

FORMER PARTICIPANTS' PROGRESS IN ENG LISH LANGUAGE CLASSROOMS 
This was the first year of the project, so there were no former participants. 
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OVERALL EDUCATIONAL PROGRESS ACHIEVED THR OUGH PROJECT 
Familiarity with American Citizenship and Culture 

While the project posed no objectives in this area, it provided a number of field 
trips for participants to acquaint them with various aspects of life in the United 
States. These trips included visits to the Statue of Liberty, Ellis Island, the 
Metropolitan Museum of Art, the Museum of the Moving Image, the Museum of 
Modern Art, and the Nicholas Roerih Museum. Participants also went on full-day 
trips to Cold Springs and to Washington, D.C. 
Career Awareness 

• Seventy-five percent of the participating students will demonstrate 

awareness of career choices and professional requirements as measured 
by a student assessment form. 

OREA-developed student surveys were translated into Russian to measure 
students' awareness of career choices and professional requirements. OREA 
received 297 completed surveys from the project. Of those, 249 (83.0 percent) 
indicated that they had developed a greater awareness of career choices and 
professional requirements us a result of participating in the project. 

The project met its objective for career awareness. 
Attitude Toward Russian Language 

• Seventy-five percent of the students will develop a more positive attitude 
toward the Russian language. 

OREA-developed student surveys were translated into Ru-sian to measure 

students' attitudes toward using the Russian language. Of the 297 completed 

surveys OREA received, 91.6 percent indicated that the students completing them 
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had gained a more positive attitude toward the Russian language after participating 
in the project. 

The project met its objective for attitude toward the Russian language. 
Grade Retention 

The project did not propose any objectives for grade retention. None of the 
participating students were retained in grade. 
Graduation 

At least 75 percent of the project students who were enrolled in their senior 
year will graduate from high school at the end of the year. 
Dropout Prevention 

The project did not propose any objective for dropout prevention. Two 
students (0.3 percent of participants) dropped out of school. 
Attendance 

• The attendance rate of the project students will be equal to that of the 
school as a whole. 

The attendance rates of project students were appreciably higher than those of 
mainstream students. (See Table 7.) 

The project met its attendance objective. 
Placement in Gifted and Talented Program 

Students identified as gifted and talented were encouraged to participate in 
appropriate activities, such as the Olympiada, Bilingual Oratory Olympics, and 
special science projects. They were offered literary assignments. Paraprofessionals 
and peer tutors worked with several of these students on individual projects. 
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TABLE 7 
Attendance Rates, by Site 



1 


Attendance Rates 


Site 


rroject oiuaenxs 




1 AKrahom I tncnln 


31.5 


OA A 

84.4 


PronUlin VS RnnQAUAlt 

rrcuiKiin L/.nuuaovou 


94.0 


87.6 


V/CU I ICO IV 10^41 Owl 1 


96.1 




Now Utrecht 


96.0 


87.6 


Lafayette 


88.7 


82.8 


1 South Shore 


92.6 


86.8 | 


Edward Murrow 


94.9 


89.1 


Forest Hills 


98.1 


85.6 


Richmond Hill 


92.6 


86.4 


Harama 


95.1 


83.2 


Be'er Hagola 


95.7 


88.3 



• Attendance rates of participating students were between 5.8 and 12.5 
percentage points higher than those of mainstream students at the 
same sites. 



CASE HISTORY 

M. was bom in Kiev, Ukraine, in 1976. He arrived in the United States in 1992 
and enrolled at Franklin D. Roosevelt High School in the fall. At that time, he had no 
English skills. Initially intimidated and confused, he soon made excellent progress 
under the guidance of project staff. M. jumped three levels in E.S.L and two levels 
in mathematics. In four months, he was able to write essays and express himself 
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easily in English. His attendance record was perfect, and ail of his grades were in 
the 90s. 

STAFF DEVE1 , OPMENT OUTCOMES 

The project did not propose any staff development objective for the year under 
review. The project provided workshops on bilingual education on a monthly basis. 

CURRICULUM DEVELOPMENT OUTCOMES 

The project did not propose any curriculum development objective for the year 
under review. Project staff, however, did develop curriculum material for Global 
History and Biology I. 

PARENTAL INVOLVEMENT OUTCOMES 

The project director reported that parents met regularly with project staff in 
individual advisement sessions. Parental involvement was a major component of the 
project, but staff found it difficult to stimulate participation. Parents did, however, 
meet the project's three objectives for their involvement: 

• Seventy percent of parents of participating Russian LE.P. students will 
have an improved involvement and understanding of their children's 
educational needs and career goals as result of participating in advisory 
council meetings, bilingual workshops, City University of New York 
bilingual education courses, or individual advisement sessions offered by 
project staff. 

Project staff administered an OREA-developed Likert-scale questionnaire to 
evaluate whether parents felt they had improved their understanding of their child's 
educational needs and career goals. Of the 199 completed questionnaires returned 
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to OREA, 165 (82.9 percent) indicated an improved understanding of their child's 
educational needs and career goals as a result of participating in the project. 

The project met its objective for improved parental understanding of student 
needs and goals. 

• At least five parents of Russian LEP students will take some of the six 
college credits required in order to serve as bilingual paraprofessionals. 

Five parents of project students registered during the spring semester at 

Adelphi for college credits toward becoming Russian bilingual paraprofessionals. 

The project met its objective for parent enrollment in college courses. 

• At least 30 parents and/or teachers, as measured by attendance records, 
will attend at least one of three workshops the project will offer on careers 
in bilingual education. 

The project sponsored four workshops. These had an average attendance of 
32 parents and teachers. 

The project met its objective for attendance at bilingual career workshops. 
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IV. SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS 



ACHIEVEMENTS OF OBJECTIVES 

The project met all of its objectives: E.S.L; Russian N.L.A.; student 
performance on Regents Competency Tests (R.C.T.s) and Regents examinations; the 
content area subjects of mathematics, science, social studies, and business; career 
awareness; attitude toward Russian language; attendance; and parental involvement. 

Participating students showed academic progress. All of the 623 participating 
students in grades nine through twelve were promoted to the next grade or 
graduated. Students showed sizable gains in English language proficiency, as 
measured by the LAB, and gains in native language arts, as indicated both by final 
course grades and the Russian language Regents examination. 

Project services not only benefited the students academically but also 
increased their awareness of the importance of education: the attendance rate of 
participating students was appreciably higher than that of the mainstream population 
at all sites. Students' awareness of their career and professional goals and 
requirements increased, and they gained a greater respect for the Russian language. 

Teachers attended monthly workshops provided by the project to increase their 
knowledge of bilingual education. 

Parents of participating students attended college courses aimed at becoming 
bilingual paraprofessionals. Workshops proved useful to parents in improving their 
involvement in and understanding of their children's educational needs and 
career goals. 
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MOST AND LEAST EFFECTIVE COMPONENTS 

Almost all components of the program were highly effective. The entire 
instructional component was particularly noteworthy. Quantitative data, reports of 
observers, and the feedback from those administering the program attested to the 
strength shown in this area 

Least effective components of the project were caieer development and 
parental involvement. Career development programming did not provide students 
with sufficient exposure to the business world, and while parental activities were 
excellent, the number of parents who took advantage of them was small. The 
project director stated that there will be increased efforts next year to involve more 
parents in the venous activities available to them. 

RECOMMENDATIONS TO ENHANCE PRO JECT EFFECTIVENESS 

• Explore additional techniques to increase students' acquisition of skills in 
mathematics, science, and social studies at those sites where a smaller 
percentage than expected passed their courses in these areas. Consider 
augmenting a peer- and paraprofessional-tutoring program to work with 
students during study halls or outside school hours. Consider also 
initiating a program of computer-assisted instruction. 

• Explore ways to develop ties with business and community organizations 
and enlist their support in the career program component as stated in the 
proposal. 

• Consider holding some parental involvement activities at sites closer to the 
homes of potential participants and at times more convenient to their 
schedules. 
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APPENDIX A 
Instructional Materials 



E.S.L 



fir** A A 

taraoe 




Author 


Publisher 


Date of 

Publication 


9-12 


Turning Point 


lantomo & Papa 


Eddison Mesdler 


1987 


intermediate 


On Target 


J. Purpura 


Scott Forosman 




Intermediate 


Variations 


P. Duffy 


Regents/ 
Prentice Hall 




9 


In Contact 1 


Raht, Deaman, 
Levie, Brigg 


Scott Foresman 




Mixed 


Great American 
Short Stories 


Draper 


Prentice Hall 


1985 


Advanced 


Contact U.SA 


A. Malkey 


Prentice Hall 





Advanced 


Grammar A & B 


Azar 


Prentice Hall 




Advanced 


Immigrant Course 


M. Cucchiara 






| Advanced 


English tor 
Changing the 
World 


C. Banks 


Scott Foreman 


1994 



N.LA. 



Grade 


Titto 


Author 


Publisher 


Date of 

Publication 


10 


Povesti 1 Rasskazi 


L Tolstoy 


Sobetskaya Rossia 






Russkfy Yazik 


A. Dudnikov 


Vishsaya Shkola 






Russian Intermediate 
Reader 


Mihalchenko 


National Text Book 
Company 




I 10 


One Day In the Life of 
Ivan Denisovich 


A. Zdzhenitsin 


YMCA - Press 
Paris 


1991 I 




Gulag Archipelago 


A. Solzhenlstin 


YMCA - Press 
Paris 
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Instructional Materials, cont'd. 



Mathematics 



| Grade 


Title 


Author 


Publisher 


Date of 
Publication 


1 

10 


Mathematics for the 
Real World 


Jack Price 


C. Merrill 
Publication 




9-12 


Integrated 
Mathematics 


Dressier & Keenan 


AMSCO 
Publication, Inc. 


1989 J 



Science 



I Grade 


Tito 


Author 


Publisher 


Date of 
Publteatibrt 


9 


Biology & Human Progress 


S. Rosen 


Global Book Company 


1988 




Biology-Comprehensive Text 


Schraer & Stone 


Cebco Standart Pub. 






The World We Live In 


S. Nomovitz 


Van Nostrant & Co. 


1965 




A Study of Changing Plan* 


B. Daley 


Allen & Bacon 






Earth Science-Brief Review 


J. Callister 


Prentice-Hall 




I 9-12 


Concepts in Modern Biology 


0. Kraus 


Globe Book Company 


1984 I 



SociaJ Studies 



1 Grade 


.TWe> ;:r- 


Author 


Publisher 


Date of 

PubBcatiori 


9 


Exploring Non- 
Western World 


Schartz & O'Connor 






Mixed 


Foundation in 
American History 


H. Schwartz 


Globe Book Company 


1987 


Mixed 


Global Studies 


H. Brum 


AMSCO Pub. 


1988 


12 


Exploring American 
Citizenship 




Globe Book Company 


1981 
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APPENDIX B 
Class Schedules 
9th Grade 



n™ Days 


Period 


Subject 1 


[ M-F 


8:45 - 9:20 


Language Workshop | 


I M " F 


9:20 - 9:55 


Global Studies 1 1 


I M-F 


9:55 - 10:30 


E.S.L J 
Communication Skills | 


M-F 


10:30 -11:05 


E.S.L. Basic English 


M-F 


11:05 -11:40 


Sequential Math III | 


LUNCH 


I M-F 


12:15 -12:50 


Physical Education 


1 M-F 


12:50 - 1:25 


Bilingual Biology 1 



10th Grade 



I Days 


Period 


Subject | 


M-F 


8:45 - 9:20 


Russian Literature 1 


M-F 


9:20 - 9:55 


Bilingual Global 
History 1 




M-F 


9:55 - 10:30 


E.S.L. 2 


M-F 


10:30 - 11:05 


E.S.L. 2 


M-F 


11:05 - 11:40 


Sequential Math II 


LUNCH 




12:15 - 12:50 


Health Education 


I M " F 


12:50 - 1:25 


Bilingual Biology 1 


| M-F 


1:25 - 2:00 


Biology Lab. 1 
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APPENDIX B 
Class Schedules, cont'd. 

11th Grade 



I Days 


Period 


Subject | 


I M " F 


8:45 - 9:20 


Business Education 


I M " F 


9:20 - 9:55 


N.LA. 


I M " F 


9:55 - 10:30 


Social Studies 


I M-F 


10:30 - 11:05 


Advanced E.S.L | 


I M-F 


11:05 - 11:40 


Advanced E.S.L | 


| LUNCH I 


I M-F 


12:15 - 12:50 


Sequential Math | 




12:50 - 1:25 


Health Education | 


12th Grade 


| Days 


Period 


Subject I 


M-F 


8:45 - 9:20 


N.LA. 


M-F 


9:20 - 9:55 


Social Studies 


M-F 


9:55 - 10:30 


E.S.L Transitional 


M-F 


10:30 - 11:05 


E.S.L Lab 


M-F 


11:05 - 11:40 


12th Year Math 


LUNCH 


M-F 


12:15 - 12:50 


Chemistry Lab 


M-F 


12:50 - 1:25 


Chemistry I 


M-F 


1:25 - 2:00 


Physical Education 



9 

ERIC 
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APPENDIX C 

BILINGUAL, MULTICULTURAL, AND EARLY CHILDHOOD EVALUATION 

OFFICE OF RESEARCH. EVALUATION, AND ASSESSMENT 

NEW YORK PUBLIC SCHOOLS 

110 LIVINGSTON STREET, ROOM 732 

BROOKLYN, N.Y. 11201 

(718) 935-3790 



PARENT QUESTIONNAIRE 
Spring 1993 

Program: SPECIAL COMPETITION FOR RECENT ARRIVALS 



Directions : Please write the numbers that show how you feel in the boxes on 
the right. 



How much has attendance at any of the following improved your 
understanding of your child's needs and career goals? 



Program's advisory committee meetings? 

Hat not improved my I I I I I I Has improved my 

understanding at all 1 2 3 4 5 understanding a lot. 



Parental workshops? 



It hasn't I I I I I I Helped 

halpad at all 1 2 3 4 5 a lot. 



3. Program's individual assessment session? 



Has not improved ay | | I I I I Has improved ay 

understanding at all 1 2 3 4 5 understanding a lot. 

Thank you for ooapletlng thia fora. 
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APPENOIX C 



BILINGUAL MULTICULTURAL, AND EARLY CHILDHOOD EVALUATION 

OFFICE OF RESEARCH. EVALUATION, AND ASSESSMENT 

NEW YORK PUBLIC SCHOOLS 

110 LIVINGSTON STREET, ROOM 732 

BROOKLYN. N.Y. 11201 

(718) 935-3790 




Program: Project SPCOM 



Student Assessment 
Spring 1993 



5. 



6. 



ERIC 



4 3^ 



Directions : Please write the numbers that show how you feel in the boxes on the 
right. 



How has Project SPCOM changed the way you feel about using Russian in your classes? 



I feel worse 
about it 



I feel better 
about it 



1 2 3 4 5 

Being able to read and write in Russian has helped me to do better in school. 



It hasn't 
helped at all 



1 2 3 4 5 



It has 

helped a lot 



3. Project SPCOM helped me keep in closer contact with my Russian culture. 



Did not help 
me at all . 



1 2 3 4 5 



Helped me 
a lot 



□ 



□ 



□ 

10 



4. By keeping in contact with my Russian culture, I will learn more about myself. 



« 



12 3 4 



Very true 



Not true ' 

at all i 1 2 3 4 5 

Because of Project SPCOM, I am more interested in continuing my education next year, 



l_l_J_J_ 
12 3 4 



I am much more 
interested 



I am NOT more 

interested 1 2 3 4 5 

Project SPCOM has helped me to do better in school. 

Did not help |_J_J_J__I__I Helped me 

me at all 1 2 3 4 5 a lot 

Project SPCOM has helped me understand the need to prepare for my future at 

college or in a career. 

Did not help | | | I I I He'P^ me 

me at all 1 2 3 4 5 a lot 
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□ 

11 

□ 

12 

□ 

13 



□ 

14 



